(1) Field of the Invention
The present invention relates to an image generation apparatus that generates a video sequence, and particularly to an image generation apparatus that generates, from a plurality of video sequences, a new video sequence having all the excellent properties of the plurality of video sequences.
(2) Description of the Related Art
With the development in digital imaging devices, they have been capable of capturing still images with high spatial resolution at relatively low cost. However, there is a problem to be solved in order to obtain images which move smoothly (for example, at approximately 30 frames per second) while maintaining the high spatial resolution equivalent to such a high spatial resolution of still images, and to further increase image capturing speed.
This is because the speed of transferring image information in an imaging device is not high enough for an enormous amount of image information obtained per unit time, and a special imaging device is needed for transferring a plurality of pieces of image information in parallel, thereby the cost of a photographing apparatus increases or a separate adjustment of the properties of the imaging device is required.
In one of the conventional image generation methods for obtaining images of a high spatial and temporal resolution, morphing is performed based on the detected corresponding points between the frames of a video sequence captured by a high-speed and low-resolution camera and a video sequence captured by a low-speed and high-resolution camera (see, for example, Japanese Patent Application Laid-Open No. 2003-203237 (FIG. 13). FIG. 1 is a diagram for explaining the conventional image generation method described in Japanese Patent Application Laid-Open No. 2003-203237. Note that in the specification of the present application, “high speed” (or “low speed”) as an imaging capability or a property of a video sequence means a relatively high frame rate (or a relatively low frame rate), and “high resolution” (or “low resolution”) means a relatively high spatial resolution (or a relatively low spatial resolution).
In FIG. 1, a high-speed and low-resolution camera A01 and a low-speed and high-resolution camera A02 photograph the same object at the same angle in synchronization with each other. FIG. 2 shows the relationship between the images captured by the camera A01 and the images captured by the camera A02. In FIG. 2, the frames captured by these two cameras A01 and A02 are shown in time order. Frames B01 and B02 are the images captured by the camera A02, while frames B11 to B15 are the images captured by the camera 01. In this diagram, the difference in spatial resolution is represented by the sizes of the images, and the frames B11 to B15 have a smaller number of pixels and thus a lower spatial resolution than that of the frames B01 and B02. On the other hand, since the frame rate of the camera A01 is higher than that of the camera A02, the camera A01 captures four frames while the camera A02 captures one frame. The camera A01 captures the frames in synchronization with the frame capturing by the camera A02 (i.e., the frames B01 and B11 are captured at the same time, and the frames B02 and B15 are captured at the same time). A method for generating a high resolution intermediate frame image which corresponds to the frame B23 in terms of position (and to the frame B13 in terms of capturing time) will be described below as an example.
As for the frame images captured as mentioned above, a primary matching unit A03 calculates the correspondence of pixels between adjacent frames in the video sequence captured by the high-speed camera A01. Here, the correspondence is the relationship between a position of a part of an object represented by each pixel in a frame image and a position of the corresponding pixel in another frame image. Next, by combining the correspondences between respective adjacent high-speed images, the primary matching unit A03 calculates the correspondence between each pixel of the high-speed camera frame B13 located at the time of the frame B23 to be generated and each pixel of the high-speed camera frame B11 captured in synchronization with the low-speed camera frame.
Next, a secondary matching unit A04 combines the positional relationship between the pixels of the low-speed camera frame B01 and the high-speed camera frame B11 captured in synchronization with each other, the positional relationship between the pixels of the frame B13 and the frame B23, and the correspondence between the frame B11 and the frame B13 calculated by the primary matching unit A03, so as to determine the correspondence of the pixels between the frame B01 and the frame B23.
Next, an image generation unit A05 determines the pixel values of the frame B23 using the pixel values of the frame B01 based on the correspondence of the pixels between the frame B01 and the frame B02, so as to generate the high-resolution intermediate frame image B23. The image generation unit A05 generates other intermediate frames in the same manner so as to generate a high-resolution and high-speed video sequence.
The other related documents (Toru MATSUNOBU, et al., “Generation of High Resolution Video Using Morphing”, Technical Report of IEICE, PRMU 2004-178, and Kiyotaka WATANABE, et al., “Generation of High Resolution Video Sequence from Two Video Sequences with Different Spatio-temporal Frequencies”, Information Technology Letters (FIT 2004), Vol. 3, No. LI-004, 2004) also show the similar methods for generating high-resolution intermediate frame images using a video sequence captured by a high-speed and low-resolution camera and a video sequence captured by a low-speed and high-resolution camera, i.e., by estimating motions between frames of the high-speed video sequence and then morphing the frames of the low-speed and high-resolution video sequence in accordance with the motions.